Skip to content

Conversation

@paulojmdias
Copy link
Member

@paulojmdias paulojmdias commented Oct 23, 2025

Description

When the log lines are written exactly with the first 1000 bytes equal, if a copytruncate happens, is there a possibility of stopping reading the file. This was discovered using the otlpjsonfile receiver and doing the copytruncate option on multiple services.

This PR resets the offsets if the current offset is higher than the file size, which usually happens when a copytruncate operation occurs.

Link to tracking issue

Fixes #43693

Testing

Added tests to simulate writing the same lines more than 1000 bytes multiple times, and validate copytruncate offsets reset.

@paulojmdias paulojmdias marked this pull request as ready for review October 23, 2025 21:00
@paulojmdias paulojmdias requested review from a team and andrzej-stencel as code owners October 23, 2025 21:00
Signed-off-by: Paulo Dias <[email protected]>
…ector-contrib into fix/43693

Signed-off-by: Paulo Dias <[email protected]>
Copy link
Member

@andrzej-stencel andrzej-stencel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm marking this as "Request changes", because I want to be sure we understand the consequences before merging this.

In principle, the fingerprint is the identifier of a file. The biggest issue I see here is that this seems to dismiss this principle in some cases.

I'll examine this pull request and your comments @paulojmdias on the corresponding issue in more detail. Thank you for digging into this! 🙏

@paulojmdias
Copy link
Member Author

Sure @andrzej-stencel! Please let me know of you need something.

Thanks for your review 🙏

Copy link
Member

@andrzej-stencel andrzej-stencel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change has the following consequence: Removing anything (e.g. the last line) from the end of a file that is being read by the File Log receiver would cause the file to be read in its entirety again, as it would be seen as a new file by the receiver. For one, this change of behavior would be a big breaking change, and for another I'm not convinced that this is the expected behavior. The fingerprint of the file stays the same, so it's the same file. We should not ingest the same file twice.

@paulojmdias
Copy link
Member Author

I've implemented the configurable solution for handling copytruncate rotation as suggested by @andrzej-stencel here. I added a new on_truncate configuration option with three modes to control behavior when a file is detected with the same fingerprint but a smaller size. The ignore mode (default) keeps the original offset and doesn't read any data until the file grows past it, maintaining current behavior and preventing duplicate log ingestion. The read_whole_file mode resets the offset to 0 and reads the entire file from the beginning, ensuring no data loss. The read_new mode updates the offset to the current file size after truncation, allowing only new data to be read.

The truncation detection logic appears in three separate code paths in file.go because the fileconsumer can encounter truncated files at different stages. The open files path handles files actively read in the previous poll cycle. The archive path handles files that haven't been seen for several poll cycles when polls_to_archive is configured. Each path retrieves file metadata differently, so the truncation check needs to happen in all three places to ensure consistent behavior, but please let me know if I'm wrong.

I added comprehensive tests for all three modes and updated the README files for both filelogreceiver and
otlpjsonfilereceiver with detailed explanations of the new configuration option.

@atoulme I'm also proposing being a CODEOWNER with you in the otlpjsonfilereceiver, but please let me know if you agree with 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collector stops reading new file after copytruncate rotation

2 participants